Non-rigid 3D shape tracking from multiview video

In most cases authors are permitted to post their version of the article (e.g. in Word or Tex form) to their personal website or institutional repository. Authors requiring further information regarding Elsevier's archiving and manuscript policies are encouraged to visit: Keywords: 3D shape tracking Mesh deformation 3D scene flow Shape from silhouette 3D video a b s t r a c t We present a fast and efficient non-rigid shape tracking method for modeling dynamic 3D objects from multiview video. Starting from an initial mesh representation, the shape of a dynamic object is tracked over time, both in geometry and topology, based on multiview silhouette and 3D scene flow information. The mesh representation of each frame is obtained by deforming the mesh representation of the previous frame towards the optimal surface defined by the time-varying multiview silhouette information with the aid of 3D scene flow vectors. The whole time-varying shape is then represented as a mesh sequence which can efficiently be encoded in terms of restructuring and topological operations, and small-scale vertex displacements along with the initial model. The proposed method has the ability to deal with dynamic objects that may undergo non-rigid transformations and topological changes. The time-varying mesh representations of such non-rigid shapes, which are not necessarily of fixed connectivity, can successfully be tracked thanks to restructuring and topological operations employed in our deformation scheme. We demonstrate the performance of the proposed method both on real and synthetic sequences. 3D modeling of dynamic real scenes is an emerging research field with applications in various domains such as 3D television, free viewpoint video, virtual reality and computer animation [1,2]. Unlike optical motion capture systems which are widely used in computer animation applications [3], 3D video methods aim to recover the complete shape of a dynamic object, not only its motion. Most of the techniques addressing the dynamic object model-ing problem adhere to passive surface reconstruction methods exploiting silhouette, shading and/or stereo information acquired from multicamera video sequences [4–13], due to the limitations of active reconstruction methods in temporal axis [14]. The goal of dynamic scene modeling schemes is usually to generate a sequence of meshes each of which represents the geometry of a dynamic object at the corresponding video frame. There are three major challenges involved in achieving this goal. The first two of these challenges concern efficiency: computational complexity of the reconstruction method and the resulting representation load. …